CS 229 Supplemental Lecture notes

Building off of our interpretations of supervised learning as (1) choosing a representation for our problem, (2) choosing a loss function, and (3) minimizing the loss, let us consider a slightly more general formulation for supervised learning. In the supervised learning settings we have considered thus far, we have input data x ∈ R and targets y from a space Y . In linear regression, this corresponded to y ∈ R, that is, Y = R, for logistic regression and other binary classification problems, we had y ∈ Y = {−1, 1}, and for multiclass classification we had y ∈ Y = {1, 2, . . . , k} for some number k of classes. For each of these problems, we made predictions based on θx for some vector θ, and we constructed a loss function L : R×Y → R, where L(θx, y) measures the loss we suffer for predicting θx. For logistic regression, we use the logistic loss L(z, y) = log(1 + e) or L(θx, y) = log(1 + e T ).